feat(proxy): attribution labels for savings, stats and export - #4699

PR governance

This PR follows the template and is marked ready for human review.

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.28571% with 34 lines in your changes missing coverage. Please review.

📢 Thoughts on this report? Let us know!

@JerrettDavis thanks for the review. Both points are fixed in the latest commit, and the branch is rebased now that #4698 has merged (it no longer carries that PR's commits).

1. Replacement resolver at the outcome funnel. emit_request_outcome now overlays the legacy client / project fields only while the dimension is enabled and still resolved by its built-in, so a replace=True resolver's label is what reaches by_dimension, the ledger's dims, the request-log tags and OutcomeSnapshot.attribution. The WebSocket and Codex handlers that bind those two labels follow the same rule. Legacy bookkeeping stays separate: the projects totals and the ledger event's own client key still record what the proxy detected. test_a_replaced_resolver_is_not_overridden_by_the_legacy_field replaces both client and project, takes the labels the middleware bound on a real request, and checks /stats, /stats-lifetime, the log tags and the ledger event.

   Getting project right needed one more change: by_dimension.project was served from the projects map, so it could not show a replaced or client-named project, and it ignored -project. project is now booked like every other dimension. projects itself is unchanged.

2. Open labels and registered names. labels now accepts a registered dimension whose resolver was not selected, so resolve_attribution(..., ['labels']) with project=web returns it, and it reaches /stats. A selected resolver keeps precedence over the header, including when it resolves nothing, and a name removed with -name is never re-admitted, by a client or by a handler.

   I did not open this to every registered name. A dimension an extension registers may come from authenticated identity, and a client should not be able to supply that one through a header just because the operator left it off the list. register_dimension therefore takes client_settable (default off): the built-ins, which are self-reported already, are client-settable, and an extension's dimension stays reserved unless it opts in. Documented and tested both ways. If you would rather have all registered names reserved, or all open, that is a small change.

Smaller things fixed along the way: unregister_dimension restores a built-in that had been replaced; default, none and labels are rejected as dimension names; and X-Headroom-Labels is filtered before the 16-label cap is applied, so names that are dropped no longer crowd out valid ones.

Two new review points on the current head: (1) aggregate_savings(..., group_by=["calls"]) loses a calls=infra label because _ranked merges the numeric calls field over it; please choose a collision-free row representation or reject conflicting dimension names and cover this case. (2) Please reconcile the replacement-client ledger guarantee: docs promise the resolved label in dims, but _event_dims drops it and the new test asserts that absence. Keeping the legacy client key is understood; the question is whether the resolved label is also retained or the guarantee narrowed. Isolated probes in /job/output/review_probes.py reproduce both; full pytest could not start because pytest is absent. No live proxy behavior claimed.

@JerrettDavis thanks, both are addressed in the new commit (a plain fast-forward on top of the head you reviewed). It also contains a set of other fixes, all listed below so nothing in the diff is a surprise.

Your two points

1. group_by=["calls"]. A row is {<dimension>: <label>, **figures}, so a dimension named like a figure cannot be grouped by. I chose to reject rather than change the row shape, which headroom savings --json consumers would see: aggregate_savings raises ValueError for any group_by name in ROW_FIELDS (calls, tokens_saved, cost_usd, …), and the CLI turns that into a --by usage error. The label is still recorded and --where calls=infra still filters. Tests: test_ledger_refuses_to_group_by_a_name_that_is_a_row_field (every field name), test_savings_cli_rejects_by_a_row_field.

2. Replacement client and the ledger. I narrowed the guarantee rather than widen the ledger. An event has one client key, always the harness, and a client label is never written to dims; the docs now say exactly that, and that --by client / --where client=… report the harness even when the resolver is replaced. Every other dimension, a replaced project included, is written to dims as resolved. Keeping a second client under dims would have given --by client and --where client= two possible meanings.

Everything else in the commit

Prompted by those two, I went through the rest of the contract the same way and fixed what did not hold.

Access

• /stats stripped savings.by_dimension for callers outside the metadata gate but still returned it under persistent_savings. Both are stripped now. test_network_callers_do_not_see_labels asserts the label string is absent from the whole /stats and /stats-lifetime bodies.

Operator switches

• Labels a handler supplies on RequestOutcome.attribution were booked even for a dimension that is off or removed with -name. They now obey the same switches as bind_label (test_a_handler_cannot_book_a_dimension_that_is_off).
• default, none and labels could still become dimensions through bind_label, a merge, or an explicit dimensions= list. They cannot now (test_spec_words_cannot_be_bound_or_merged_as_dimensions).
• Request-log client / project tags stay what the proxy detected unless the operator listed the dimension, so a label a client merely names through labels does not rewrite them (test_client_named_labels_do_not_change_the_request_log_tags).

Label handling

• A label was percent-decoded again at each stage, so a%252541 was stored differently by the context, the snapshot and the tracker. It is decoded once, where the header is read (test_a_header_label_is_percent_decoded_exactly_once). One contract change follows from that: resolve_attribution no longer decodes what a resolver returns, so a resolver that reads a header itself decides whether to decode it. sanitize_label gained decode= for this. The built-ins are unaffected.
• Only the first 4 KB of X-Headroom-Labels is read, since it is parsed on every request (test_only_the_start_of_the_labels_header_is_read).

Bounds

• The tracker's 16 dimension slots were first come, first kept, for good. A new dimension now evicts the smallest, oldest idle one, the same rule as labels (test_a_new_dimension_evicts_the_smallest_idle_one).
• Merged labels are capped at 16, with later layers (what a handler knows) keeping their place (test_merge_keeps_the_later_layers_when_over_the_cap). bind_label does not add a 17th (test_bind_label_stops_at_the_cap).

aggregate_savings and headroom savings

• A --where or --by name that is not a usable dimension name was dropped silently, giving an unfiltered report. It is now a ValueError / usage error (test_ledger_rejects_names_that_are_not_dimensions).
• group_by given as a bare string is one name, not its characters.
• A --where label is cleaned the same way stored labels are before comparing.
• SavingsReport.dimensions lists every dimension in the retained events, as its comment says; it was computed after the filter (test_ledger_lists_every_dimension_whatever_the_filter).
• Repeating --where on one dimension is an error (it kept only the last); a filter that matches nothing says so and names the filter; --by client prints one client section, not two (test_savings_cli_filter_edges).
• --by help no longer names host and traffic_class, which arrive in the next PR.

Smaller

• Built-in resolvers are installed under a lock and the flag is set afterwards, so a concurrent first request cannot see a partial registry.
• The dashboard table sorts rows by tokens saved itself; integer-like labels such as 42 otherwise jumped to the top.
• test_a_bad_spec_warns_once_not_per_request asserted <= 1 warnings and could pass with none; it now clears the cache and asserts exactly one.

Docs (attribution.mdx, security-model.mdx)

• --where applies to every window; --by splits the lookback window only.
• The 16 / 50 caps are the proxy's lifetime totals; the ledger keeps each event's labels for its retention window.
• none stops labels being resolved and booked; the ledger's own client key is still written.
• unknown covers any event with no label for the dimension, not only old ones.
• -name removes from what the rest of the list selects, so -client alone selects nothing; write default,-client.
• The gate covers persistent_savings.by_dimension too, and /stats-history carries by_dimension behind its own operator-only access.
• project has its own dashboard table, read from the older projects totals.

Not covered

The proxy tests hand the labels the middleware bound to emit_request_outcome directly, so there is no single test that drives a model request from middleware to outcome through a handler.

A remaining formatter case changes attribution labels: from headroom.proxy.attribution import format_labels, parse_labels; print(parse_labels(format_labels({'team': 'release%41'}))) prints {'team': 'releaseA'}. Expected the valid literal label release%41 to survive the advertised inverse operation; actual behavior silently puts it in a different bucket because parsing percent-decodes it. The separator rejection fix does not cover percent escapes. Please choose and test an explicit formatter policy (encode literal percent signs, or reject/document such values). This was reproduced in an isolated Python unit probe; no live proxy claim.

@JerrettDavis thanks, fixed in the new commit (a plain fast-forward on top of d13f4b92).

Percent escapes in format_labels

I went with encoding. parse_labels splits on , and ; and then percent-decodes each label once, so format_labels now percent-encodes exactly those three characters in a label: % → %25, , → %2C, ; → %3B. Every label that normalize_attribution keeps now survives the round trip:

>>> format_labels({'team': 'release%41'})
'team=release%2541'
>>> parse_labels(format_labels({'team': 'release%41'}))
{'team': 'release%41'}

This also removes the earlier separator rule. A label containing , or ; used to be dropped with a warning; now it is encoded and sent (design, ops → team=design%2C ops). Parsing is unchanged, so headers that clients already send mean what they meant before.

Tests in tests/test_attribution.py:

• test_format_labels_roundtrip_keeps_percent_and_separators, parametrized over release%41, 100%, %%, %2541, design, ops, a;b, k=v and 50% off; ,x. Each must come back unchanged next to a second label.
• test_format_labels_encodes_only_what_the_parser_reads pins the exact header text.

On the previous head, 6 of these cases fail (release%41, %2541, 50% off; ,x, design, ops, a;b and the exact-output test). All of them pass with the fix.

Test evidence

Run in a sealed container (Python 3.13, no network, sources copied in, nothing mounted) on the tree of the new commit:

• Every test file that mentions attribution (40 files, including test_attribution.py, test_attribution_savings.py, test_extension_attribution.py, test_pipeline_outcome_stage.py, test_proxy_savings_history.py and test_prometheus_label_escaping.py): 963 passed, 2 skipped.
• ruff check and ruff format --check on both changed files are clean.

The three flagged check runs

Run 131027444949 (template), run 131027444646 (label) and run 43172051543 (PR Governance) on d13f4b92 were cancelled, not failed. Each was cancelled and then passed on a rerun of the same head, which is listed in the checks. The new push runs them again.

@JerrettDavis thanks, good catch. Fixed, together with a few more things we found reviewing the rest of the PR ourselves. Everything is a fast-forward: a merge of upstream/main (it had started to conflict), then one commit per fix.

The header limit

LABELS_HEADER_MAX_LENGTH is now computed from the other limits instead of being a fixed 4096:

LABELS_HEADER_MAX_LENGTH = MAX_DIMENSIONS * (DIMENSION_NAME_MAX_LENGTH + len("=") + 3 * LABEL_MAX_LENGTH) + (MAX_DIMENSIONS - 1) * len("; ")
# 16 * (32 + 1 + 384) + 15 * 2 = 6702

That is the longest header format_labels can build from ASCII labels: every name and label at full length, with every character percent-encoded. So the parser reads anything the formatter builds in full, and the limit follows if the other limits ever change.

The parser also stops shortening labels. When a header is longer than the limit, it is cut back to the last separator, so a pair cut by the limit is dropped instead of being booked under a shortened label (your d10 case). That applies to headers clients build by hand too.

format_labels now also percent-encodes non-ASCII, so the header is plain ASCII. A non-ASCII character can encode to up to 12 characters, so for those labels the formatter raises ValueError instead of building a header the parser would cut.

Tests (tests/test_attribution.py):

• test_format_labels_reported_overflow_case_roundtrips: your exact reproduction.
• test_format_labels_longest_output_is_read_whole: the longest possible header is exactly the limit and comes back unchanged, through parse_labels and resolve_attribution.
• test_a_pair_cut_by_the_header_limit_is_dropped_not_shortened: the boundary on both sides, including a separator exactly at the limit.
• test_format_labels_encodes_non_ascii (café, 東京, emoji, mixed) and test_format_labels_refuses_a_header_the_parser_would_cut.

Other fixes from our own review

1. Deferred gateway outcomes were booked under another request's labels. With relay_usage, the outcome is recorded later: by the response half, or by whichever request's register()/get() sweeps the expired turn. Both run in a different request's context, so that request's labels and conversation got the savings. A PendingTurn now keeps a copy of its request half's context and the draft is recorded in it. An orphaned draft is recorded with no labels. Tests: test_an_expired_draft_is_booked_under_its_own_requests_labels, test_an_orphaned_draft_is_booked_with_no_labels and, end to end through /v1/compress/response, test_deferred_outcome_is_booked_under_the_request_halfs_labels.
2. Per-label token totals left out deferred tool schemas. by_dimension counted their dollars but not their tokens, so a label could show $0.06 next to 0 tokens, disagree with headroom savings --by, and be the first one evicted. It now has tool_tokens_saved and headline_tokens_saved, as by_model does, and ranks and evicts by the sum. The dashboard card uses the sum. Test: test_a_label_counts_deferred_tool_schemas_like_the_ledger.
3. Labels were trimmed before they were cut to 128 characters, so a space at position 128 survived. Cleaning the result again gave a different label, which could split one label into two counters across a restart. They are now cut first and trimmed second. Test: test_a_label_cut_to_length_keeps_no_trailing_space.
4. The conversation id was percent-decoded twice (abc%2541 became abcA). set_current_conversation now takes the already-decoded value. Covered in test_snapshot_falls_back_to_the_client_named_conversation.
5. The labels switch used up one of the 16 dimension slots, so as the 17th entry it was dropped without notice. It no longer counts against the limit. Test: test_the_labels_switch_does_not_take_a_dimension_slot.
6. Dimension names starting with - or . are refused. A client could create -project through labels, but an operator could never list it, because -x means "remove x". Tests: test_sanitize_dimension_name and test_a_name_starting_with_a_dash_cannot_be_sent_as_a_label.
7. unknown is not accepted as a label. The ledger already shows it for "no label", so --where host=unknown mixed the two together. Test: test_unknown_is_not_a_label.
8. Removed the HEADROOM_LABELS constant. It was documented as something headroom wrap reads, but nothing read it.
9. Docs: the header limit and the label rules above, the per-label token fields, and a correction. A replaced resolver's label wins in request-log tags, but when it finds nothing the tag still shows what the proxy detected.

Merge with upstream

upstream/main (fda75de5) conflicted in two places, both from #4859:

• /stats: fix(proxy): withhold per-project savings and quota stats from /stats network callers #4859 now hides per_project, projects and the quota sections from network callers; this PR hides by_dimension. The merge keeps both.
• The security-model.mdx table: the rows list both sets of fields.

Test evidence

Everything ran in a sealed container (Python 3.13, no network, sources copied in, nothing mounted), on the tree of the new head:

• Broad run: every test file that mentions attribution, per_project or /stats-lifetime, plus tests/gateway/ and test_gateway_turn_unit.py (60 files): 1753 passed, 6 skipped, 1 failed.
• The one failure is tests/gateway/test_compress_turn_seam.py::test_contract_disabled_by_env (502 where it expects 404/405). It fails the same way on the previous head 9c4f2f1f in the same container, and it does not touch attribution.
• Before the fix: with the previous head's code and the new tests, 20 of the new tests fail.
• Each commit: built one at a time on the previous head, each commit passes test_attribution.py, test_attribution_savings.py, test_gateway_turn_unit.py and gateway/test_contract_response_half.py.
• Lint: ruff check and ruff format --check are clean on every changed file.